Shiny for R-Package Risk Assessment
Aaron Clark, 16 Aug 2023
Consortia of ~50 companies (mostly pharma and biotech)
Focus on designing/building a framework to validate R and R packages with an eye to all open-source software.
The main goal is ensuring proper validation documentation exists in accordance with expectations of regulatory agencies.
Checkout pharmar.org to learn about all work streams and get involved!
Two tools: what do they do
is a framework to quantify an R package’s “risk” by assessing several meaningful metrics designed to evaluate package development best practices, code documentation, community engagement, and development sustainability.
is a full-fledged R package containing a shiny front-end that augments the utility of {riskmetric}. The application’s goal is to provide a central hub for an organization to review and assess the risk of R packages, providing handy tools and guide rails along the way.
Quantify risk programmatically
Sometimes “quality” is measurable! Software dev best practices dictate an R-package should have:
- A license
- Source code available for browsing
- An easy to contact maintainer
- A place to report bugs
- Evidence that new bugs are being addressed
- Complete Function documentation
- Adequate test coverage
- Community usage
18 total assessments (to date)!
Why create a Shiny app?
Main goal: help those those making “package inclusion” requests for GxP environments. So, the app’s highest and best use is all about two things:
- Forcing org members to take responsibility for assessing package risk themselves prior to making an IT request
- Generating a summary report for IT that shows the requester has done their due diligence meeting inclusion requirements
Why create a Shiny app? (cont’d)
Main goal: help those those making “package inclusion” requests for GxP environments. They need support to…
- Provide a platform for package exploration without the need to write any custom
{riskmetric} - Run
{riskmetric}on the same machine with the same environment – creating a central hub for reproducibility - Maintain consistent, org-specific settings/options when producing risk outputs
- Automate a risk-based “decision triage” based on pre-defined rules, saving time & effort
- Manage who’s involved in the review process via user authentication & role management
- Facilitate and store user summaries & communication, on certain packages and/or certain metrics
- Generate risk summary reports, for sharing with the decision making parties
Latest features (v2.0.0)
Most notably:
- Face lift to the
'Report Builder'&'Database View' - Better support to analyze dependencies
- More org-level customization, including the use of a configuration file
- Allow
adminusers to edit roles and privileges on the fly - Allow users to explore source contents of package
The feedback loop is crucial! Submit an issue on GitHub today.
Latest: 'Report Builder' Face Lift
A more holistic Report Builder! Allowing users to define what content shows up in the report.
Users can now compose a long form “Package Summary” to keep track of more pertinent items (perhaps non-
{riskmetric}items) for a more rounded package review.
Latest: 'Database View' Face Lift
More useful content to ingest!
- A summary of uploaded packages
- Date uploaded
- Decision-related columns like the decision time stamp & decision source
- Easily downloadable
Latest: Package Dependencies
Latest: org-level settings
Edit in-app or via the config file! What’s new?
- Customize decision categories & colors
- Toggle decision automation rules
- User roles/privileges
- Initialize metric weights
Example config (YML) file:
default:
assessment_db: database.sqlite
loggit_json: loggit.json
credential_db: credentials.sqlite
credentials:
roles:
- admin
- lead
- reviewer
- viewer
privileges:
admin: [admin, weight_adjust, auto_decision_adjust, final_decision, revert_decision, add_package, delete_package, overall_comment, general_comment]
lead: [weight_adjust, auto_decision_adjust, final_decision, revert_decision, add_package, delete_package, overall_comment, general_comment]
reviewer: [add_package, general_comment]
viewer:
decisions:
categories:
- Low Risk
- Medium Risk
- High Risk
rules:
Medium Risk:
- .398
- .639
High Risk:
- .639
- 1
colors:
Medium Risk: !expr grDevices::rgb(52, 235, 229, maxColorValue = 255)
metric_weights:
covr_coverage: 0
has_vignettes: 2
Automated a risk-based “decision triage” based on pre-defined rules, saving time & effort
Decisions impacted upon pkg upload:
Latest: Edit Roles & Privileges
Manage who’s involved in the review process via user authentication & role management
Latest: Explore Package Source Code
Augment {riskmetric} evaluation with a more manual package review process by browsing source code
Latest features recap (v2.0.0)
Most notably:
- Face lift to the
'Report Builder'&'Database View' - Better support to analyze dependencies
- More org-level customization, including the use of a configuration file
- Allow
adminusers to edit roles and privileges on the fly - Allow users to explore source contents of package
The feedback loop is crucial! Submit an issue on GitHub today.
An exercise: Review the {prodlim} package
Demo on Shinyapps.io!
Test drive now: bit.ly/raa_demo
Coming soon
{riskscore}
In it’s early stages, a new data package containing the {riskmetric} assessments & scores for all packages on CRAN.
Coming soon
Package Explorer
A suite of hands-on review tools (from our friends at GSK) will allow users to browse unit tests, source code, and man (doc) pages for any exported function.
Dev Team
Q & A
bit.ly/raa_demo
bit.ly/raa_gh
bit.ly/raa_survey
pharmar.org/
